RESEARCH & PRACTICE IN ASSESSMENT 



Notes in Brief 

There are likely as many approaches to teaching assessment as there are 
people teaching assessment. Graduate courses on assessment can be structured 
with a singular focus, such as learning outcomes assessment, or along a 
competencies-based framework. Such frameworks include the Assessment Skills 
and Knowledge (ASK) Standards developed by College Student Educators 
International (ACPA) in 2006 and the ACPA/NASPA Professional Competencies 
(Bresciani & Todd) introduced in 2010 and revised in 2015, which include the 
Assessment, Evaluation, and Research competency area. The purpose of this 
article is to share reflections on an approach to teaching assessment through the 
use of a CAS Standards (Council for the Advancement of Standards in Higher 
Education, 2012) self-study in a master’s level assessment course during the Fall 

2015 semester. 
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M. here are likely as many approaches to teaching assessment as there are people 
teaching assessment. Since first developing an assessment course in 2009 I have approached 
teaching assessment in a few different ways. I first used the Assessment Skills and 
Knowledge (ASK) Standards developed by College Student Educators International (ACPA) 
(Mitchell, 2006). I have tweaked, added, and subtracted instructional approaches, activities, 
and assignments in the course with the introduction of the ACPA/NASPA Professional 
Competencies (Bresciani & Todd) in 2010, which included the Assessment, Evaluation, 
and Research competency area. The purpose of this article is to share reflections on a new 
approach I took using the CAS Standards (Council for the Advancement of Standards in 
Higher Education, 2012) in a master’s level assessment course during the Fall 2015 semester. 

One of the required courses in the master’s graduate preparation program at Murray 
State University is institutional research and assessment. The purpose of the course is to 
help students develop core competencies related to assessment work in postsecondary 
institutions. The learning outcomes from the course include understanding the development 
and evolution of the institutional research and assessment functions, identifying research 
needs within an institution, preparing assessment plans, and gathering, analyzing, and 
synthesizing data from multiple sources. 


While on the faculty at a previous institution, I designed the assessment course 
CORRESPONDENCE so q lat students would work with campus partners to carry out assessment projects—but 

that came with mixed results. At times, some campus partners had not previously engaged 
Email in assessment efforts and others had expectations beyond what students learning the 
bbourke@murraystate.edu process themselves could realistically provide. With those past experiences in mind, I set 

out to develop a different hands-on project for students to learn how to gather, analyze, and 
interpret data, and then synthesize and report findings. 
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The hands-on project that students completed during the Fall 2015 semester was an analysis 
of the master’s graduate preparation program at Murray State University. The program analysis 
project was based on the GAS Standards for Master’s Level Student Affairs Administration 
Preparation Programs (Council for the Advancement of Standards in Higher Education, 
2012; referred to as Master’s Program GAS Standards throughout the remainder of this 
article). Students enrolled in the course served as the review team conducting the analysis 
of the master’s program. The Master’s Program GAS Standards are divided into nine parts: 
program mission and objectives; recruitment and admission; curriculum policies; pedagogy; 
curriculum; equity and access; academic and student support; professional ethics and legal 
responsibilities; and program evaluation. 

CAS Standards 

The Council for the Advancement of Standards in Higher Education (GAS) 
is a consortium of 42 professional associations with the aim of advancing the use of 
professional standards for the purpose of continuous quality improvement of programs 
and services (Council for the Advancement of Standards in Higher Education, 2015). 
Continuous quality improvement is addressed through the following goal: “to promote 
the assessment and improvement of higher education services and programs through 
self-study, evaluation, and the use of CAS standards” (Council for the Advancement of 
Standards in Higher Education, 2015, “CAS Purpose” p.2). The Master’s Program CAS 
Standards reflect one of the 45 sets of standards. 

The precursor to the current version of the CAS Standards for master’s programs 
was first published in 1979 as “Standards for the Preparation of Counselors and College 
Student Affairs Specialists at the Master’s Degree Level” (CAS, 2013, p. 2). The Master’s 
Program CAS Standards, last revised in 2012 (CAS, 2012), consist of nine parts: mission and 
objectives, recruitment and admission, curriculum policies, pedagogy, curriculum, equity 
and access, academic and student support, professional ethics and legal responsibilities, and 
program evaluation. As with the other CAS Standards, the master’s program standards are 
supplemented by a Self-Assessment Guide (SAG). The purpose of the SAG is to provide a 
systematic and standardized approach to identifying strengths and weakness through a self- 
study (CAS, 2013). Each SAG includes a section titled “Formulating an Action Plan,” which 
consists of guiding questions provided to facilitate discussions aimed at enacting plans for 
improvement (CAS, 2013). 

Steps in the self-study 

While CAS does not prescribe a static procedure for conducting a self-study, there is 
a set of five recommended steps listed in the SAG (CAS 2013): 

1. Form the review team. The recommendation is to form a team with 
diverse perspectives, including a chair and other members from outside 
of the unit under study. 

2. Prepare and train the review team. To prepare, team members should 
familiarize themselves with the standards appropriate for the review and 
come to a consensus for interpreting information and generating ratings. 

3. Compile and review documents and other sources of evidence. In addition 
to evaluating documents and other data, review team members might seek 
out other sources of data following sharing scale ratings with staff from the 
unit under study (CAS, 2016). For the master’s program review additional 
sources included conducting interviews with various stakeholders, including 
faculty, current students, and administrators. 

4. Review documents and other evidence of program performance. The fourth 
step consists of conducting the review, through which the review team will 
use the standards criteria statements and assign a rating to each one, using 
the scale provided to reflect degree of compliance (from Does Not Meet to 
Exceed). Generally, team members do this individually and then meet to 



compare ratings, discuss and resolve discrepancies, and finalize their 
collective evaluation. (GAS, 2016, p. 7) 

5. Write up review results and recommendations. The recommendations 
mav be as specific as setting “a timetable for addressing deficiencies” 

(CAS, 2013). 

Assessment Course Project 

To reflect the course-level learning outcome of learning how to gather, analyze, and 
synthesize data, I provide an account through the rest of this article of the procedures that the 
review team (students enrolled in the course) utilized toward this aim. Note that the steps we 
took as a learning exercise varied slightly from those suggested for a self-study as described 
by in the Master’s Program GAS Standards SAG (2013). Because the self-study was adapted as 
an instructional activity, significant emphasis was placed on the second and third steps in the 
self-study process. 

The students were provided the SAG developed for the Master’s Program CAS 
Standards. We spent time reviewing and discussing the steps of what is termed as the self- 
assessment process during each weeldv class session. The time spent each week served as 
the first two steps in the process—which are to establish and prepare the review team and to 
understand the GAS standards and guidelines of the self-study. We discussed and developed 
plans for completing the third step, which involves compiling and reviewing documentary 
evidence. As I share in the section on analyzing data, we used class meeting time to demystify 
the fourth step of judging performance, based on compiled evidence. Throughout the semester 
we discussed the fifth and final step of completing the CAS self-assessment process. This 
last step involves examining individual and group ratings assessed in the fourth step and 
synthesizing the review team’s evaluation of the extent to which the master’s program meets 
each CAS Standard. 

Gathering data 

One of the themes I stressed in the course was that good assessment work relies on 
multimodal data collection and analysis. I avoided calling this mixed methods research, as 
utilizing varied approaches to organizational effectiveness, student learning, or other common 
assessment aims does not reflect a cogent mixed methods research design (Creswell, 2015). 

As a class, the students identified key stakeholders who could offer perspectives 
and provide data via interview. Each student assumed responsibility for interviewing a key 
stakeholder and then transcribing the interview. The transcripts were posted to a shared 
online file-sharing space for easy access. The transcripts were used to answer the questions 
associated with each SAG part. Students were also expected to identify data needs prior 
to conducting the interviews so that they could ask for further documentation from each 
stakeholder. All documents and other data gathered in this manner were also posted to the 
shared online file-sharing space. 

Analyzing data 

Data analysis presented a bit of a challenge. Although an introduction to research 
methods is a prerequisite for the assessment course students did not feel confident in their 
abilities to analyze data. As we engaged in the CAS review process, the students saw the 
process as being more qualitative and subjective. Their concerns and trepidations were with 
the prospect of manipulating SPSS or other statistical software, but the students became much 
more comfortable with the idea of data analysis for the CAS review as the course progressed. 
Students’ concerns were further eased as I guided them through analyzing data for Part 1: 
mission and objectives. As noted in the previous section, data from various sources, including 
interview transcripts, were used to answer the questions from each part of the SAG. 

The primary component of the CAS self-study is the use of the rating scales for each 
of the nine parts. In order to complete the ratings for the items listed for each part in the SAG 
students gathered and evaluated evidence prior to determining rankings on individual items. 
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For example, when addressing Part 5: the curriculum, students examined program documents 
that described the curriculum, reviewed course syllabi, and read transcripts from interviews 
with stakeholders. 

Synthesizing findings 

To synthesize the findings, the students created a format based on the nine parts of the 
SAG that was used for the semester-long project. In class, we discussed pulling together data 
from multiple sources related to the same points of inquiry: the numeric rating items and the 
summary questions associated with each part within the SAG. The students experienced the 
challenge of synthesizing the basis for each numeric rating along with addressing information 
from transcripts of interviews with stakeholders. 

During class discussions, we talked about the challenge of acknowledging subjective 
biases when attempting to report in a seemingly objective manner. The challenge rested 
in evaluating something that they were in the process of experiencing: their own graduate 
preparation program. Through our class discussions I emphasized the importance of looking 
at multiple data points in order to arrive at numeric ratings. As we addressed each of the 
parts of the SAG I asked students to mark their ratings based on their own experiences and 
perceptions. Then students put those ratings aside and attempted to make their ratings based 
on the data that had been collected. More often than not students’ data-based ratings varied 
from those recorded from their personal experiences and perceptions. 

Reporting findings/results 

The students synthesized their findings and produced a 30-page report. This extensive 
report was structured based on the parts of the SAG and each section consisted of an item-by¬ 
item breakdown of the numeric ratings with a summary of the analysis that led to the rating 
of each item. 

The final report that the students generated was shared with program faculty, and 
the department chair. The students expressed concern about being identified in the event 
there were items in the report that were (or perceived to be) negative. I addressed this by only 
including the course number and semester on the report. With this step taken the students 
indicated that they felt they could be honest in writing up the report— in the event of any 
negative findings. However, students remained anxious about the possibility of backlash in the 
event of negative findings due to the small number of students in a single section of the course. 
Their worries stemmed from their position as students and the power differential between 
themselves and the stakeholders interviewed as part of the data collection process. 

Lessons Learned 

The lessons learned from the use of the GAS Standards for a program self-study that 
are addressed in this section are focused on programmatic efforts and not directly on student 
learning. The self-study was the graduate program’s first foray into formalized assessment and 
helped to establish a foundation for a culture of assessment. The institution requires student 
learning outcomes assessment but overall program evaluation is not required. The faculty 
wanted to capitalize on opportunities to assess the graduate program because it is new but 
also wanted some form of a baseline to guide future assessment efforts. The student-written 
report has led program faculty to develop a more extensive and comprehensive assessment 
plan that goes beyond learning outcomes assessment as mandated by the institution. The plan 
includes the continuation of the self-study as part of the assessment course, alumni surveys, 
benchmarking of comparable graduate programs at peer institutions, and data on internship 
and job placements. 

In its initial offering, six students were enrolled in the course. With a small number 
of students in the course I was able to divide the nine parts of the SAG among students and 
also had students collaborate on some of the parts of the SAG that were more labor intensive. 
I was concerned about workload and did not have students extend institutional comparisons 
beyond what was available through peer-program websites and graduate program directories. 
Through the GAS self-study students demonstrated learning on multiple fronts. Students 
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learned a hands-on approach to assessment via self-study. The biggest learning takeaway 
students demonstrated was the collection, evaluation, and reporting based on multiple sources 
of evidence. As demonstrated in their written reports, students analyzed data from documents, 
interviews, and institutional data to draw conclusions and make recommendations. 

In the future, with more students enrolled, I will divide the parts differently so that 
students are gaining experience in gathering, analyzing, and interpreting data from multiple 
sources, across multiple parts of the SAG, as well as synthesizing and reporting findings. One 
area I did not address in this first attempt was to report findings in varied formats and for 
varied audiences. While we discussed various reporting formats in the course, students did not 
gain direct experience. 


Conclusion 

When I began writing this article my initial intent was to reflect on the use of the 
GAS Standards as a tool for teaching an aspect of assessment. By shifting my reflection to 
the form of a publication I engage in a key aspect of the scholarship of teaching and learning 
that Shulman (2001) labels as professionalism. As a member of professional and scholarly 
communities I have a responsibility to share what I learn through teaching (Shulman, 2001). 
By sharing my reflections from aspects of a course that I teach not only do I share what I have 
learned but I am also making my teaching available for public view and critique (Ginsberg & 
Bernstein, 2011). 

The approaches I took in using the GAS Standards in a graduate-level course do 
not have to be exclusive to formal courses. Similar approaches can be taken in concert with 
efforts to build and sustain cultures of assessment (see Gulp & Dungy, 2012). A GAS self- 
study can serve as a great tool for staff within a department to learn aspects of assessment 
and evidence-based decision making. I have seen a GAS self-study process modified to be 
conducted completely by within-unit staff as a precursor to a review by an external team. 
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